swadhin / awesome-db-roadmap public
/ awesome-db-roadmap / 04 - Big Data.md
swadhinbiswas ✨ Initial commit: Ultimate Database Roadmap 🗂️🚀 - From Zero to Pro! 📚💡 · 11/20/2025
e598c7b
44 lines · 1.3 KB
Raw Download

04 – Big Data & Analytics

When data volume, velocity, or variety exceed a single node, we enter the big data world.

📌 Overview

  • Understand the difference between OLTP vs OLAP systems
  • Learn the building blocks of large-scale storage and analytics
  • See how data flows from source → storage → processing → consumption

🧱 Core Components

[[Distributed Storage]]

  • Learn why we need distributed file/object stores (HDFS, S3, GCS)
  • Understand replication, erasure coding, and data locality

[[Data Warehouses]]

  • Understand columnar storage and MPP architectures
  • Compare on-prem vs cloud warehouses (Snowflake, BigQuery, Redshift)

[[Query Engines]]

  • Learn about engines that sit on top of storage (Presto/Trino, Hive, Spark SQL)
  • Understand separation of storage and compute

[[Streaming Systems]]

  • Understand event streams and log-based architectures (Kafka, Pulsar)
  • Learn about streaming processing engines (Flink, Spark Streaming)

🔗 Connections

  • Align with [[01 - Fundamentals]] for replication, sharding, and consistency
  • Connect with [[07 - Data Modeling]] for warehouse schemas (star/snowflake)
  • Relate to [[05 - Cloud Databases]] for managed big data services